Tests are commonly used to verify whether generic systems (and especially software applications) behave as expected. In this context, many approaches are available to test the software applications.
For example, a complete test is run when a new software application (or the first version of a new level thereof involving major changes) is released. In this case, the test must have a very high coverage, so as to ensure that most features of the software application are validated (for their correctness, completeness and quality). For this purpose, a suite of test cases is executed; each test case involves the application of a predefined input to the software application, which returns a corresponding output in response thereto; a result of the test case is determined by comparing the actual output provided by the software application with an expected response thereof.
On the other hand, a regression test may be used for a new version, release or service level of the software application—typically involving minor changes to a part thereof only. The regression test is aimed at ensuring that the unchanged parts of the software application still work correctly—i.e., that none of the corresponding test cases already executed “regressed” from a positive result (when the test case passed) to a negative result (when the test case failed).
The regression test allows verifying that the changes applied to the software application have not inadvertently introduced any errors into the other parts thereof. For example, this may occur because some preexisting features of the software application do not work properly with the added features; another problem may be due to undiscovered errors (in the preexisting features) that are brought to light by the added features.
However, the regression test is a very time consuming activity. Particularly, in large software applications each run of the regression test typically involves the execution of thousands of test cases (which may require several working days to complete). This drawback is magnified in very dynamic environments, wherein the same operations must be repeated at any change of the software application.
In order to solve the problem, selective strategies are commonly used. The selective strategies (contrary to the so-called retest-all strategy) attempt to reduce the number of test cases to be executed by skipping the ones that are unlikely to be affected by the applied changes. As a consequence, it is possible to avoid executing test cases unnecessarily. This allows maintaining the cost of the regression test at a reasonable level.
Several criteria have been proposed for selecting the test cases to be regressed (among all the ones being available). For example, coverage-based criteria discard the test cases exercising components of the software application that are independent of the applied changes.
However, this requires a thoughtful knowledge of the relationship between the test cases and the components of the software application; at the same time, it is necessary to know the impact of the applied changes (both on the components being directly impacted and on the ones correlated thereto). Therefore, the selection of the test cases involves a deep investigation activity, which must be performed substantially in a manual way by one or more testers. The problem is further exacerbated when the complete test and the regression test are entrusted to different teams, so that the persons in charge of the regression test are unaware of how the complete test was defined.
Therefore, the design of the regression test requires a heavy human intervention. As a consequence, the obtained results are not repeatable and prone to errors (being strongly dependent on the skill of the testers).
A solution for automating the selection of the test cases to be regressed is disclosed in US-A-2004/0154001. This document proposes the definition of a profile that allows identifying the test cases that are likely to invoke one or more modules of the software application being impacted by the applied changes. However, this requires instrumenting the software application with extra code intended to collect statistics that are then recorded in the profile.